refactor(series)!: 🕰️ drop TimestampSeries #1274

cmp0xff · 2025-07-13T06:40:48Z

Closes Series + Timedelta = no overload variant of ... #1355
Closes str + Series -> Never #1359
Addresses CLEAN: Investigate whether TimestampSeries, TimedeltaSeries, etc. can be removed #718
Tests added: Please use assert_type() to assert the type of any return value

tests/test_timefuncs.py

pandas-stubs/core/series.pyi

tests/test_series.py

tests/test_frame.py

tests/test_scalars.py

Dr-Irv · 2025-07-23T16:53:15Z

@cmp0xff you have a number of PRs submitted while I was out on vacation for 2 weeks. Can you let me know which ones I should prioritize for review?

cmp0xff · 2025-07-23T18:15:12Z

Hi @Dr-Irv, I hope you had a nice vacation. My pull requests are categorised below. Each category is independent, but those in a higher position have a slightly higher priority in my opinion.

`Series`: arithmetic operations

The following two PRs are independent. They migrate test_series.py to a subfolder series, and add quite a few test_*.py files there.

add: feat(series): #1098 arithmetic addition #1275
truediv: feat(series): #1098 arithmetic truediv #1280

`DataFrame.to_dict`

fix(DataFrame): #799 to_dict #1283

`Index.append`

feat(index): append #1282

`Series`: address #718

refactor(series)!: 🕰️ drop TimestampSeries #1274 - this is a prerequisite for the next one.
refactor: #718 also drop TimedeltaSeries #1273

Dr-Irv

Thanks for doing this. It's a lot of good work.

Main thing - if I'm going to merge this PR, it needs to be in a state where we don't need the followup PR.

Basic rule - we don't put ignore in the tests unless we are testing that the stubs should not accept something that is invalid. You have places where you have added ignore in the tests and I won't merge that in (unless we know it is a bug in the type checker)

docs/philosophy.md

tests/test_frame.py

tests/test_scalars.py

tests/test_timefuncs.py

pandas-stubs/core/series.pyi

Dr-Irv · 2025-07-24T21:28:34Z

Hi @Dr-Irv, I hope you had a nice vacation. My pull requests are categorised below. Each category is independent, but those in a higher position have a slightly higher priority in my opinion.

I've reviewed them all, except #1273 as noted there.

Thanks for all the great work.

cmp0xff · 2025-07-24T22:45:42Z

I've reviewed them all, except #1273 as noted there.

Thanks for all the great work.

Thank you very much for your quick and thorough reviews. I will be able to work on them next week.

…les#r2229555145

…74/files#r2229550572

…les#r2229581983

cmp0xff · 2025-09-04T21:46:26Z

Yes, but I think we can get around this by never having a type corresponding to Series[Any] using the PUnknown idea I listed above.

If I understand correctly, we can do the experiment below: (it's an pyi file, instead of py file, so no implementation is needed)

from typing import Any, overload, Generic, reveal_type
from typing_extensions import TypeVar, Never, Self

class PUnknown: ...

T = TypeVar("T", bound=int | PUnknown, default=PUnknown)

class Se(Generic[T]):
    @overload
    def __sub__(self: Se[int], other: Se[int]) -> Never: ...
    @overload
    def __sub__(self, other: Self) -> Self: ...

def foo(a: Se[int]) -> Se[int]: ...

reveal_type(foo(Se[PUnknown]()))  # mypy, pyright: cannot assign

reveal_type(Se[PUnknown]() - Se[PUnknown]())  # mypy, pyright: Se[PUnknown]

I think it will be a big change, worthy a separate PR, if we do it. In particular, foo(Se[PUnknown]()) gives an assignment error now, because int is not a subtype of PUnknown. In the current approach, in contrast, int is a subtype of Any.

Dr-Irv · 2025-09-04T23:27:42Z

I think it will be a big change, worthy a separate PR, if we do it. In particular, foo(Se[PUnknown]()) gives an assignment error now, because int is not a subtype of PUnknown. In the current approach, in contrast, int is a subtype of Any.

I played with the idea. Seems like a lot of work. The subtype argument is what makes it a problem. So let's not go down that path.

Above, mypy indeed gives Any. However pyright gives Unknown. If following this documentation is the right way to go, pyright is deviating from it. Moreover, the first and the third overloads are indeed incompatible. Removing any one of them will leads to a resulting typing of the other one.

I actually don't think that pyright is wrong by saying Unknown. It's saying it couldn't make a match that would resolve the type. I like that pyright says Unknown vs. Any because Any is different (and can be declared in a type declaration), whereas Unknown says that the code is ambiguous.

I wish the typing spec allowed you to have Unknown as a type that was treated differently with respect to generics.

What I want to argue is that Series[Any] - Series[Timestamp] -> Never and Series[Any] - Series[Any] -> Series[Any] are incompatible, if we follow the documentation of the python typing team.

I see your point. So where does that leave us? What are our options?

This might be the argument to keep TimestampSeries and TimedeltaSeries ?? One way to look at it is that for Index we have DatetimeIndex and TimedeltaIndex, so maybe TimestampSeries and TimedeltaSeries are analogous?

cmp0xff · 2025-09-06T22:34:40Z

So where does that leave us? What are our options?

The following is my understanding:

We can keep TimestampSeries and TimedeltaSeries.
- Pro: We are able to keep Series[Any] - TimestampSeries -> Never, etc, while keeping Series[Any] - Series[Any] -> Never
- Con: TimestampSeries and TimedeltaSeries are quite unintuitive. Even as a contributor to pandas-stub, who knows about TimestampSeries, I am still reluctant to look up how to import TimestampSeries, when I just need to cast. It would be much easier to be able to use pd.Series[pd.Timestamp] for users.
We can drop TimestampSeries in this PR, and keep Series[Any] - Series[Timestamp] -> Never just for pyright.
- Pro: Our philosophy is kept within pyright
- Con: mypy does not agree. Furthermore, as we have discussed below, being able to keep Series[Any] - Series[Timestamp] -> Never violates the rules described by the python typing team, which pyright possibly needs to fix in the future.
We can drop TimestampSeries in this PR, as well as make Series[Any] - Series[Timestamp] -> Series[Any].
- Pros:
  - Results are consistent for pyright and mypy
  - Results are consistent with simpler types. The python native types, Any - datetime.datetime, gives Any instead of Never. If pandas-stubs follows this example, Series[Any] - Series[Timestamp] -> Series[Any] seems legitimate to me.
```
from typing import Any, reveal_type
from datetime import datetime

import pandas as pd

a: Any

reveal_type(a - datetime(2025, 1, 1))  # Any
reveal_type(a - pd.Timestamp(2025, 1, 1))  # Any
```
- Con: we need to change our philosophy.

Dr-Irv · 2025-09-09T05:02:36Z

We can drop TimestampSeries in this PR, and keep Series[Any] - Series[Timestamp] -> Never just for pyright.

Pro: Our philosophy is kept within pyright

Con: mypy does not agree. Furthermore, as we have discussed below, being able to keep Series[Any] - Series[Timestamp] -> Never violates the rules described by the python typing team, which pyright possibly needs to fix in the future.
We can drop TimestampSeries in this PR, as well as make Series[Any] - Series[Timestamp] -> Series[Any].
Pros:
Results are consistent for pyright and mypy
Results are consistent with simpler types. The python native types, Any - datetime.datetime, gives Any instead of Never. If pandas-stubs follows this example, Series[Any] - Series[Timestamp] -> Series[Any] seems legitimate to me.
from typing import Any, reveal_type
from datetime import datetime

import pandas as pd

a: Any

reveal_type(a - datetime(2025, 1, 1))  # Any
reveal_type(a - pd.Timestamp(2025, 1, 1))  # Any
Con: we need to change our philosophy.

The choice is between (2) and (3). I'm not sure with (2) how to handle the pyright vs. mypy difference. If we could have a simple example that we bring up with them and see how they debate it, I'd be willing to let that play out to see which type checker decides to change the way they are handling things. Is that something you could do? (I think I tried, but couldn't get anything simple to work).

With respect to (3), I think the issue here is Any vs. Unknown, which pyright differentiates, while mypy does not (as best as I can tell). So for DataFrame.__getattr__(), we are declaring that as returning Series, pyright sees that as Series[Unknown] and mypy sees it as Series[Any]. So your example in (3) isn't the same, because generics are getting in the way.

Having said that, you're saying "we need to change our philosophy", so can you be more clear on how you would document that?

Note - I see you requested another review. I'm traveling, so can't do that for a few days, but we need to resolve this discussion first anyway.

cmp0xff · 2025-09-11T20:24:23Z

Hi @Dr-Irv ,

The choice is between (2) and (3). I'm not sure with (2) how to handle the pyright vs. mypy difference. If we could have a simple example that we bring up with them and see how they debate it, I'd be willing to let that play out to see which type checker decides to change the way they are handling things.

I think my previous example can do it. Let me make it even shorter (note it is a .pyi, not a .py):

from __future__ import annotations
from typing import (
    Any,
    overload,
    Generic,
    Never,
    reveal_type,
    Self,
)
from typing_extensions import TypeVar

T = TypeVar("T", int, str)

class Se(Generic[T]):
    @overload
    def __sub__(self: Se[int], other: Se[int]) -> Never: ...
    @overload
    def __sub__(self, other: Self) -> Self: ...

def t1() -> None:
    reveal_type(Se[Any]() - Se[Any]())  # mypy: Any, pyright: Never.

mypy thinks both Se[int] - Se[int] -> Never and Self - Self -> Self apply. The result is ambiguous, so it gives Any.
pyright does not take into account the second overload. However, I feel that pyright should have returned Unknown, see its documentation.

With respect to (3), I think the issue here is Any vs. Unknown, which pyright differentiates, while mypy does not (as best as I can tell). So for DataFrame.__getattr__(), we are declaring that as returning Series, pyright sees that as Series[Unknown] and mypy sees it as Series[Any]. So your example in (3) isn't the same, because generics are getting in the way.

from __future__ import annotations
from typing import (
    Any,
    Generic,
    overload,
    reveal_type,
)
from typing_extensions import TypeVar

T = TypeVar("T", int, str)

class Se(Generic[T]):
    @overload
    def __sub__(self: Se[int], other: Se[int]) -> Se[int]: ...
    @overload
    def __sub__(self, other: Se) -> Se: ...

def foo(a: Se[int]) -> Se[int]: ...

def t1() -> None:
    reveal_type(Se[Any]() - Se[str]())  # mypy: Se[Any], pyright: Se[Unknown].

The logic of pyright on Unknown is here. I think it says that when pyright cannot infer a type, it gives Unknown.

Probably, asking the pyright community can enlighten the discussion.

Having said that, you're saying "we need to change our philosophy", so can you be more clear on how you would document that?

In terms of Python script, we currently have the following in philosophy.md:

frame = pd.DataFrame({"timestamp": [pd.Timestamp(2025, 8, 26)], "tag": ["one"], "value": [1.0]})

timestamps = frame["timestamp"]
reveal_type(timestamps)  # type checker: Series[Any], runtime: Series
reveal_type(timestamps - pd.Timestamp(2025, 7, 12))  # type checker: Unknown and error, runtime: Series
reveal_type(cast("TimestampSeries", timestamps) - pd.Timestamp(2025, 7, 12))  # type checker: TimedeltaSeries, runtime: Series

tags = frame["tag"]
reveal_type("suffix" + tags)  # type checker: Never, runtime: Series

After changing our philosophy, we will have something like

frame = pd.DataFrame({"timestamp": [pd.Timestamp(2025, 8, 26)], "tag": ["one"], "value": [1.0]})

timestamps = frame["timestamp"]
reveal_type(timestamps)  # type checker: Series[Any], runtime: Series
reveal_type(timestamps - pd.Timestamp(2025, 7, 12))  # type checker: Series[Any] or Series[Unknown], runtime: Series
reveal_type(cast("Series[Timestamp]", timestamps) - pd.Timestamp(2025, 7, 12))  # type checker: TimedeltaSeries, runtime: Series

tags = frame["tag"]
reveal_type("suffix" + tags)  # type checker: Series[Any] or Series[Unknown], runtime: Series

Dr-Irv · 2025-09-11T20:54:57Z

@cmp0xff I created a pyright issue: microsoft/pyright#10924

Thanks for your analysis on how we'd be changing our philosophy if we chose option (3). The way I see it is that option (2) is providing more error checking for users than option (3). However, with option (2), you are also forced to say that df["somecol"] has a specific type (via cast) when doing certain calculations.

I'm leaning towards (3), but let's see what the pyright response is to the issue.

Dr-Irv · 2025-09-13T20:02:28Z

@cmp0xff I created a pyright issue: microsoft/pyright#10924

Thanks for your analysis on how we'd be changing our philosophy if we chose option (3). The way I see it is that option (2) is providing more error checking for users than option (3). However, with option (2), you are also forced to say that df["somecol"] has a specific type (via cast) when doing certain calculations.

I'm leaning towards (3), but let's see what the pyright response is to the issue.

So it appears that it's a known bug in pyright, which means we need to do (3). Let's go ahead with that and update the philosophy accordingly. I guess we might just need to delete the section on "Generic Series have restricted arithmetic", but maybe just some edits are needed? Or we should just say that we can't catch all invalid usage?

cmp0xff · 2025-09-13T20:49:13Z

Casting approach

One of our aims is to help the user recognising potential mistakes. When it comes to Series[Any], our philosophy and implementation in the current main, include the following, at type checking:

Series[Any] + Series[str] -> Never
Series[Any] - TimestampSeries -> Never
Series[Any] + TimestampSeries -> Never

Users are asked to cast the left operand Series[Any].

This is no more viable if we want to drop TimestampSeries, due to the typing rules of mypy and pyright.

Passive approach

We could follow the examples from native Python types. Consider the following attempt.pyi

from datetime import datetime
from typing import Any, reveal_type


a: Any
b: datetime

reveal_type(a + "test")  # Any
reveal_type(a - b)  # Any
reveal_type(a * b)  # Any

The typing is quite passive, especially in the third case, where datetime does not support multiplication at all, so that the expression necessarily fails at run time (as far as I know).

However this approach ensures maximum extensibility for subclasses.

In this approach, the examples from the casting approach become

Series[Any] + Series[str] -> Series[Any]
Series[Any] - Series[Timestamp] -> Series[Any]
Series[Any] + Series[Timestamp] -> Series[Any]

Progressive approach

This is developed from my "consistent plan" in #1343 (comment). In contrast to the casting approach, we do

Series[Any] + Series[str] -> Series[str]
Series[Any] - Series[Timestamp] -> TimedeltaSeries
Series[Any] + Series[Timestamp] -> Series[Timestamp]

The argument is, if the arithmetic works at run time, the resulting type is the only valid type. Users can be helped by the static type checking, when they see the progressively given resulting type is non-sense.

If the valid resulting type is not unique, we give Series[Any] instead, for example Series[Any] + Series[int] -> Series[Any].

Dr-Irv · 2025-09-15T13:47:36Z

Progressive approach

This is developed from my "consistent plan" in #1343 (comment). In contrast to the casting approach, we do

I'm fine with this approach. Let me know when I should review.

…les/f855f8655fcbc0c13879f404926034621641bc58#r2291133305

cmp0xff

Hi @Dr-Irv , I hope I have addressed all discussions around the stub files.

philosophy.md remains to be updated later in this PR.

tests/series/arithmetic/str/test_add.py

pandas-stubs/core/series.pyi

pandas-stubs/core/indexes/accessors.pyi

Dr-Irv · 2025-09-15T16:32:07Z

Hi @Dr-Irv , I hope I have addressed all discussions around the stub files.

philosophy.md remains to be updated later in this PR.

Let me know when you have everything updated (including philosophy.md) and then I'll do a more complete review.

cmp0xff

Hi @Dr-Irv , I also proposed a new text for philosophy in ff37521. Now the PR is complete from my point of view.

pandas-stubs/core/series.pyi

Dr-Irv

Some small things and 2 larger things:

There are some tests that you removed that should work at runtime - inconsistencies in pandas interpretation that you can subtract a single datetime, but not a list of datetimes, so I'd like you to create an issue in pandas about any of those.
There are changes here that are using Series[Timedelta], but I'd like to have this PR only deal with Series[Timestamp] but still use TimedeltaSeries whenever we have Series[Timedelta]. Then the other PR can fix that.

docs/philosophy.md

pandas-stubs/_libs/tslibs/timestamps.pyi

Dr-Irv · 2025-09-16T14:58:50Z

tests/series/arithmetic/test_sub.py

-        _0 = left_ts - s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
        _1 = left_ts - a  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
-        _2 = left_td - s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
-        _3 = left_td - a  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]


I'm a little puzzled here. We can see that Series[Any] - list[datetime] is invalid, but why can't we see that Series[Any] - datetime is invalid?

Or maybe Series[Any] - list[datetime] should be allowed??

Same comment for the reverse operation and the .sub()

I think the issue here is that pandas is inconsistent, so can you report that in pandas?

At run time,

Series[Timestamp] - list[datetime] is not implemented. ENH: arithmetic between DatetimeArray and list pandas#62353

Series[Timestamp] - datetime is valid.
from datetime import datetime from typing import assert_type import pandas as pd arr = pd.to_datetime(["2020-01-01", "2020-01-02"]).array assert isinstance(arr, pd.arrays.DatetimeArray) print(arr - datetime(2020, 1, 1))

can you put back the tests that were there ? If we don't catch them in type checking, add a comment referring to the pandas issue.

tests/series/arithmetic/test_sub.py

Dr-Irv · 2025-09-16T15:08:58Z

tests/series/arithmetic/timestamp/test_add.py

+
+    if TYPE_CHECKING_INVALID_USAGE:
+        _0 = left + s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
+        _a = left + d  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]


I think Series[Timestamp] + Sequence[timedelta] should be valid.

Pandas does not support adding a list to a DatetimeArray: pd.Series([pd.Timestamp("2025-01-01")]) + [pd.Timedelta(1, "s")] gives TypeError: unsupported operand type(s) for +: 'DatetimeArray' and 'list' pandas-dev/pandas#62353

OK - can you put a comment in here that says this should work, so we are currently detecting it as a typing error, and refer to the pandas issue?

pandas-stubs/core/series.pyi

Dr-Irv

I think the only thing that's now needed is some comments related to the things that are not working in pandas but we'd like to test. That means putting some tests back that you deleted with appropriate comments.

Ideally, if we believe it should work, but pandas says it doesn't, then we should have the type checker catch it until pandas fixes things (if ever). So I'd rather keep the tests that you deleted in tests/series/arithmetic/test_sub.py and add any appropriate comments that point to the pandas issues.

Dr-Irv · 2025-09-16T21:52:12Z

tests/series/arithmetic/timestamp/test_sub.py

+    left.sub(s)
+    left.sub(d)

-        left.rsub(s)  # type: ignore[call-overload] # pyright: ignore[reportArgumentType,reportCallIssue]
-        left.rsub(d)  # type: ignore[call-overload] # pyright: ignore[reportArgumentType,reportCallIssue]
+    left.rsub(s)


The fact that these work is part of the inconsistency. Can you add a comment that points to the pandas issue 62353 ?

Dr-Irv · 2025-09-16T21:55:01Z

tests/series/arithmetic/test_sub.py

-        _0 = left_ts - s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
        _1 = left_ts - a  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
-        _2 = left_td - s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
-        _3 = left_td - a  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]


can you put back the tests that were there ? If we don't catch them in type checking, add a comment referring to the pandas issue.

tests/series/arithmetic/test_sub.py

Dr-Irv · 2025-09-16T21:56:07Z

tests/series/arithmetic/timestamp/test_add.py

+
+    if TYPE_CHECKING_INVALID_USAGE:
+        _0 = left + s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
+        _a = left + d  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]


OK - can you put a comment in here that says this should work, so we are currently detecting it as a typing error, and refer to the pandas issue?

Dr-Irv · 2025-09-16T21:56:40Z

tests/series/arithmetic/timestamp/test_sub.py

+        _0 = left - s  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
+        _a = left - d  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]
+
+        _1 = s - left  # type: ignore[operator] # pyright: ignore[reportOperatorIssue]


Can you add a comment that describes this?

pandas-stubs/core/series.pyi

cmp0xff mentioned this pull request Jul 13, 2025

refactor: #718 also drop TimedeltaSeries #1273

Draft

2 tasks

cmp0xff marked this pull request as ready for review July 13, 2025 07:05

cmp0xff changed the title ~~fix: #718 only drop TimestampSeries~~ refactor: #718 only drop TimestampSeries Jul 13, 2025